Method For Displaying Recognition Result Obtained By Three-Dimensional Visual Sensor And Three-Dimensional Visual Sensor

ABSTRACT

Display suitable to an actual three-dimensional model or a recognition-target object is performed when stereoscopic display of a three-dimensional model is performed while correlated to an image used in three-dimensional recognition processing. After a position and a rotation angle of a workpiece are recognized through recognition processing using the three-dimensional model, coordinate transformation of the three-dimensional model is performed based on the recognition result, and a post-coordinate-transformation Z-coordinate is corrected according to an angle (elevation angle f) formed between a direction of a line of sight and an imaging surface. Then perspective transformation of the post-correction three-dimensional model into a coordinate system of a camera of a processing object is performed, and a height according to a pre-correction Z-coordinate at a corresponding point of the pre-coordinate-transformation three-dimensional model is set to each point of a produced projection image. Projection processing is performed from a specified direction of a line of sight to a point group that is three-dimensionally distributed by the processing, thereby producing a stereoscopic image of the three-dimensional model.

BACKGROUND OF THE INVENTION

The present invention is based on Japanese Patent Application No.2009-057430 filed with the Japan Patent Office on Mar. 11, 2009, theentire content of which is hereby incorporated by reference.

1. TECHNICAL FIELD

The present invention relates to a three-dimensional visual sensor thatrecognizes an object through three-dimensional measurement processingusing a stereo camera, particularly to a technology for displayingrecognition result.

2. Related Art

For example, when three-dimensional recognition processing is performedin order to cause a robot to grasp a component in a manufacturing scene,the three-dimensional information restored by the three-dimensionalmeasurement of a stereo camera is checked with the previously registeredthree-dimensional model of the recognition-target object to recognizethe position and attitude (specifically, a rotation angle with respectto three-dimensional model) of the recognition-target object (forexample, see Japanese Unexamined Patent Publication No. 2000-94374).Occasionally the similar technique is applied to inspection to determinewhether a position or an attitude of an inspection object is proper.

Generally the three-dimensional model used in the recognition processingexpresses a shape (mainly contour) of the full-scale recognition-targetobject by plural three-dimensional coordinates. In the recognitionprocessing, the three-dimensional model and restored three-dimensionalinformation are correlated to each other such that a degree ofcoincidence between the three-dimensional model and restoredthree-dimensional information becomes the maximum.

At this point, the position and rotation angle of the three-dimensionalmodel are specified as the position and attitude of therecognition-target object.

The recognition processing result can be displayed as a coordinateexpressing the position and an angle expressing the attitude. However,because only a simple numerical display is difficult to understand,there is a user demand to be able to easily confirm the recognitionresult and accuracy of the recognition result.

The inventor studies to perform coordinate transformation of thethree-dimensional model based on the recognized position or rotationangle to three-dimensionally display the post-coordinate-transformationmodel. When perspective transformation of the three-dimensional model towhich the coordinate transformation into an imaging surface of thecamera is performed based on the recognition result, a projection imagethat is seen in a way similar to that of the recognition-target objectis produced. Therefore, the inventor studies a method for changing theprojection image of the three-dimensional model to a stereoscopicdisplay according to a change in a direction of a line of sight based onthe display in which the projection image of the three-dimensional modelis superimposed on the image of the recognition-target object.

However, in the perspective transformation processing, because theprojection image is produced based on a parameter reflectingmagnification of the camera so as to be seen in the way similar to thatof the actual image, even in the plane having the same area, a size ofthe plane formed on the imaging surface depends on a distance from thecamera. Therefore, when the perspective transformation of the image inwhich a height according to a Z-coordinate at a corresponding point ofthe original model is correlated to each point of the projection imageof the three-dimensional model is performed from a direction distantfrom a direction orthogonal to the imaging surface, the projection imagewhose size varies depending on a difference of the Z-coordinate isdirectly projected to produce the image having a large deformation, andtherefore the shape of the three-dimensional model cannot properly beexpressed.

SUMMARY

The present invention has been devised to solve the problems describedabove, and an object thereof is to perform the display suitable to theactual three-dimensional model or the recognition-target object when thestereoscopic display of the three-dimensional model is performed whilecorrelated to the image used in the three-dimensional recognitionprocessing.

In accordance with one aspect of the invention, a method is performed inorder to display recognition result of a three-dimensional visualsensor, and the three-dimensional visual sensor includes: an imagingunit that includes at least one camera; a registration unit in whichthree-dimensional information expressing a full-scale recognition-targetobject is registered as a three-dimensional model; and a recognitionunit that performs three-dimensional measurement to a recognition-targetobject whose three-dimensional model is registered in the registrationunit using an image produced by imaging of the imaging unit and checksthree-dimensional information restored by the measurement with theregistered three-dimensional model to recognize a position and anattitude of the recognition-target object, on condition that a worldcoordinate system is defined such that a distance from a reference planetraversing a direction of an optical axis of the camera in the imagingunit becomes a Z-coordinate indicating a height. In the method accordingto the present the invention, at least one of the cameras is set to aprocessing object, and a first step, a second step, a third step, and afourth step are performed every camera of the processing object.

In the first step, a manipulation of specification of a direction of aline of sight with respect to an imaging surface of the camera isreceived. In the second step, plural Z-coordinates included in athree-dimensional model is corrected to perform perspectivetransformation of a post-correction three-dimensional model into acoordinate system of the imaging surface of the camera of the processingobject for the three-dimensional model to which coordinatetransformation is performed in a world coordinate system such that theposition and attitude of the three-dimensional model become identical tothose of the recognition-target object recognized by the recognitionunit. Hereinafter the perspective transformation is referred to as“first perspective transformation”. The relationship between thecoordinate system of the camera and the world coordinate system isexpressed by an equation (1). Each element of a perspectivetransformation matrix in the equation (1) is determined by previouscalibration, which allows the relationship with each coordinate systemto be specified.

In the third step, perspective transformation of a point having athree-dimensional coordinate value obtained by setting a heightaccording to a corresponding pre-correction Z-coordinate into a pointhaving a three-dimensional coordinate value obtained by setting a pointincluded in an projection image of the three-dimensional model producedin the coordinate system of the imaging surface of the camera of theprocessing object by the perspective transformation processing isperformed from the direction of the line of sight received in the firststep. Hereinafter the perspective transformation is referred to as“second perspective transformation”.

In the fourth step, the projection image produced by the perspectivetransformation in the third step is displayed on a monitor device.

In the above method, a rule of correction operation is applied to theZ-coordinate correction in the second step. In the rule of correctionoperation, a correction amount to each Z-coordinate is set to zero whenan orthogonal relationship is established between the direction of theline of sight received in the first step and the imaging surface of thecamera of the processing object, each Z-coordinate is corrected to aconstant value Z0 when the direction of the line of sight becomesparallel to the imaging surface, each post-correction Z-coordinate isbrought close to the constant value Z0 as the relationship between thedirection of the line of sight and the imaging surface becomes close toa parallel state from the orthogonal state, and a difference betweenpost-correction Z-coordinates Z1′ and Z2′ of any two of Z-coordinates Z1and Z2 (Z1>Z2) is gradually decreased while a relationship of Z1′>Z2′ ismaintained.

In the above method, after the Z-coordinate of the three-dimensionalmodel to which the coordinate transformation is already performed so asto be matched with the recognized position and attitude is correctedbased on the relationship between the direction of the line of sightspecified by the user and the imaging surface of the camera of theprocessing object, the perspective transformation of the post-correctionthree-dimensional model into the coordinate system of the camera of theprocessing object is performed (first perspective transformation).Further, the height according to the pre-correction Z-coordinate at thecorresponding point of the post-coordinate-transformationthree-dimensional model is set to each point of the projection imagethat is produced in the camera coordinate system through the firstperspective transformation, whereby the points are disposed in theposition corresponding to the height at the corresponding point of thethree-dimensional model to perform second perspective transformation tothe three-dimensionally-distributed points.

Assuming that x, y, z are axes of the coordinate system of the camera(xy-plane corresponding to the imaging surface), when the directionorthogonal to the imaging surface, that is, the direction in which theimaging surface is squarely seen is specified as the direction of theline of sight, because a correction amount of the three-dimensionalmodel to the Z-coordinate becomes zero, thepost-coordinate-transformation three-dimensional model is directlyprojected to the xy-plane (imaging surface) of the camera coordinatesystem in the first perspective transformation. Therefore, it isconsidered that the projection image produced in the xy-plane is similarto the image produced when the three-dimensional model is imaged withthe camera of the processing object. Even if the height according to theZ-coordinate at the corresponding point of the three-dimensional modelis set to each point to perform the perspective transformation from thedirection orthogonal to the imaging surface, the projection image isseen in the way similar to that of the projection image produced throughthe first perspective transformation. Accordingly, the image similar tothat of the case in which the image that thepost-coordinate-transformation three-dimensional model is projected tothe imaging surface of the camera of the processing object can beexpressed even if the Z-coordinate is not corrected.

On the other hand, when the direction parallel to the imaging surface isspecified as the direction of the line of sight, that is, when thedirection orthogonal to the z-axis direction of the camera coordinatesystem is specified as the direction of the line of sight, all theZ-coordinates are corrected to the constant value Z0. In the firstperspective transformation, the point in which the X- and Y-coordinatesare identical is projected to the same coordinates in the xy-plane ofthe camera coordinate system irrespective of the pre-correctionZ-coordinate. Accordingly, when the height according to thepre-correction Z-coordinate at the corresponding point of thepost-coordinate-transformation three-dimensional model is set to eachpoint of the projection image of the three-dimensional model, eachprojection point in which the X- and Y-coordinates are identical can bedisposed at the height according to the original Z-coordinate while thex- and y-coordinates are maintained in the same state. Therefore, theimage expressing the state in which the three-dimensional model isobserved right beside can be produced by performing the perspectivetransformation of the points from the immediately lateral direction.

When the direction oblique to the imaging surface is specified as thedirection of the line of sight, each Z-coordinate is corrected such thateach of the post-correction X-, Y-, and Z-coordinates come close to theconstant value Z0 as the direction of the line of sight is brought closeto the state parallel to the imaging surface, and such that a differencebetween the post-correction Z-coordinates is decreased. Accordingly,even if the direction of the line of sight is largely deviated from thedirection orthogonal to the imaging surface, each point of theprojection image can be disposed at the height according to the originalZ-coordinate to perform the second perspective transformation while theinfluence of the Z-coordinate on the projection image produced by thefirst perspective transformation is decreased, so that the deformationof the projection image in the second projection transformation can beprevented from being generated. Therefore, the image in which thethree-dimensional model observed from the specified direction isexpressed without a feeling of strangeness can be produced.

Accordingly, a user can visually confirm the recognition result with thethree-dimensional model by confirming the state of the three-dimensionalmodel in the projection image that is displayed while the direction ofthe line of sight is changed in various ways.

In the method of the above aspect, preferably a step of specifying asize per one pixel is previously performed to the camera of theprocessing object based on a relationship between the coordinate systemof the camera and the world coordinate system, the size per one pixelbeing allocated to the imaging surface in performing perspectivetransformation of a plane in which a height in a space is zero into theimaging surface of the camera of the processing object. In the thirdstep, for each point of the projection image of the three-dimensionalmodel, the pre-correction Z-coordinate at a corresponding point of thepre-coordinate-transformation three-dimensional model is converted intothe number of pixels based on the size per one pixel specified in thestep, and the number of pixels is set to the height after theconversion.

Accordingly, because a scale factor substantially identical to that ofthe x- and y-coordinates is set to the z-coordinate of each projectionpoint of the three-dimensional model, the size in the height directionof the projection image of the three-dimensional model can properly beset, and the image can be produced and displayed without a feeling ofstrangeness.

In accordance with another aspect of the invention, a three-dimensionalvisual sensor includes an imaging unit that includes at least onecamera; a registration unit in which three-dimensional informationexpressing a full-scale recognition-target object is registered as athree-dimensional model; and a recognition unit that performsthree-dimensional measurement to a recognition-target object whosethree-dimensional model is registered in the registration unit using animage produced by imaging of the imaging unit and checksthree-dimensional information restored by the measurement with theregistered three-dimensional model to recognize a position and anattitude of the recognition-target object, on condition that a worldcoordinate system is defined such that a distance from a reference planetraversing a direction of an optical axis of the camera in the imagingunit becomes a Z-coordinate indicating a height.

The three-dimensional visual sensor according to the present inventionalso includes a specifying manipulation receiving unit that sets atleast one of the cameras to a processing object to receive amanipulation of specification of a direction of a line of sight withrespect to an imaging surface of the camera of the processing object; afirst perspective transformation unit that corrects plural Z-coordinatesincluded in a three-dimensional model to perform perspectivetransformation of a post-correction three-dimensional model into acoordinate system of the imaging surface of the camera of the processingobject based on a relationship between the direction of the line ofsight received by the specifying manipulation receiving unit and theimaging surface of the camera of the processing object, thethree-dimensional model to which coordinate transformation is performedin a world coordinate system such that the position and attitude of thethree-dimensional model become identical to those of therecognition-target object recognized by the recognition unit; a secondperspective transformation unit that performs perspective transformationof a point having a three-dimensional coordinate value obtained bysetting a height according to a corresponding pre-correctionZ-coordinate into a point having a three-dimensional coordinate valueobtained by setting a point included in an projection image of thethree-dimensional model produced in the coordinate system of the imagingsurface of the camera of the processing object by the perspectivetransformation processing performed through the first perspectivetransformation unit from the direction of the line of sight received inthe specifying manipulation receiving unit; and a display control unitthat displays the projection image produced through the perspectivetransformation performed by the second perspective transformation uniton a monitor device. In the three-dimensional visual sensor according tothe present invention, a rule of correction operation is applied to theZ-coordinate correction in the first perspective transformation unit. Inthe rule of correction operation, a correction amount to eachZ-coordinate is set to zero when an orthogonal relationship isestablished between the direction of the line of sight received in thefirst step and the imaging surface of the camera of the processingobject, each Z-coordinate is corrected to a constant value Z0 when thedirection of the line of sight becomes parallel to the imaging surface,each post-correction Z-coordinate is brought close to the constant valueZ0 as the relationship between the direction of the line of sight andthe imaging surface becomes close to a parallel state from theorthogonal state, and a difference between post-correction Z-coordinatesZ1′ and Z2′ of any two of Z-coordinates Z1 and Z2 (Z1>Z2) is graduallydecreased while a relationship of Z1′>Z2′ is maintained.

In the recognition result displaying method and the three-dimensionalvisual sensor to which the recognition result displaying method isadopted, the user can easily confirm the recognition result by thechecking with the three-dimensional model through the three-dimensionaldisplay of the three-dimensional model, and the user-friendliness of thethree-dimensional visual sensor is considerably enhanced.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a configuration of a production line to which athree-dimensional visual sensor according to an embodiment of theinvention is introduced;

FIG. 2 is a block illustrating an electric configuration of thethree-dimensional visual sensor of the embodiment;

FIG. 3 illustrates a configuration example of a three-dimensional model;

FIG. 4 illustrates a confirmation screen of recognition result performedby the three-dimensional model;

FIG. 5 illustrates an example in which an image on the confirmationscreen is updated;

FIG. 6 illustrates an example in which an image on the confirmationscreen is updated;

FIG. 7 is a flowchart illustrating a procedure of processing forproducing a three-dimensional model image; and

FIG. 8 is a flowchart illustrating a procedure of processing performedto one workpiece.

DETAILED DESCRIPTION

FIG. 1 illustrates an example in which a three-dimensional visual sensor100 according to an embodiment of the invention is introduced to aproduction line of a factory.

The three-dimensional visual sensor 100 of the embodiment is used torecognize a position and an attitude of a workpiece W (morphology issimplified for the sake of convenience) conveyed by a conveying line 101in order to assemble the workpiece W in a predetermined product.Information indicating recognition result is transmitted to a robotcontroller (not illustrated) of a robot (not illustrated) disposeddownstream in the conveying line 101, and the information is used tocontrol an operation of the robot.

The three-dimensional visual sensor 100 includes a stereo camera 1 and arecognition processing device 2 that is disposed near the conveying line101. The stereo camera 1 includes three cameras A, B, and C that arehorizontally disposed above the conveying line 101. The central camera Ais disposed while an optical axis of the central camera A is orientatedtoward a vertical direction (that is, the central camera A squarely seesthe workpiece W), and the right and left cameras B and C are disposedwhile optical axes are inclined.

The recognition processing device 2 is a personal computer in which adedicated program is stored. The recognition processing device 2includes a monitor device 25, a keyboard 27, and a mouse 28. In therecognition processing device 2, after images produced by the cameras A,B, and C are captured to perform three-dimensional measurement to acontour of the workpiece W, restored three-dimensional information ischecked with a previously-registered three-dimensional model.

FIG. 2 is a block illustrating a configuration of the three-dimensionalvisual sensor 100.

Referring to FIG. 2, the recognition processing device 2 includes imageinput units 20A, 20B, and 20C corresponding to the cameras A, B, and C,a camera driving unit 21, a CPU 22, a memory 23, an input unit 24, adisplay unit 25, and a communication interface 26.

The camera driving unit 21 simultaneously drives the cameras A, B, and Cin response to a command provided from the CPU 22. Therefore, the imagesproduced by the cameras A, B, and C are inputted to the CPU 22 throughthe image input units 20A, 20B, and 20C.

The display unit 25 is a monitor device of FIG. 1. The input unit 24 isa device in which the keyboard 27 and mouse 28 of FIG. 1 are collected.In performing calibration processing, the input unit 24 and the displayunit 25 are used to input setting information and to display informationfor assisting work. The communication interface 26 is used to conductcommunication with a robot controller.

The memory 23 includes a large-capacity memory such as ROM, RAM, and ahard disk. Programs for the calibration processing, production of thethree-dimensional model, and three-dimensional recognition processing ofthe workpiece W and setting data are stored in the memory 23. Aparameter for the three-dimensional measurement computed through thecalibration processing and the three-dimensional model are alsoregistered in a dedicated area of the memory 23.

Based on the program in the memory 23, the CPU 22 performs thecalibration processing and processing for registering thethree-dimensional model. Therefore, the three-dimensional recognitionprocessing can be performed to the workpiece W.

In the calibration processing, using a calibration plate (notillustrated) in which a predetermined calibration pattern is drawn, aworld coordinate system is defined such that a distance from a referenceplane (that is, an upper surface of the conveying line 101 of FIG. 1)supporting the workpiece W becomes a Z-coordinate indicating a height.The imaging of the calibration plate and the image processing areperformed in plural cycles. Using a plural combinations ofthree-dimensional coordinates (X, Y, Z) of the world coordinate systemand two-dimensional coordinates (x, y) on a imaging surface of thecamera, a 3′4 perspective transformation matrix adopted to the followingtransformation equation (equation (1)) is derived every camera (S in theequation (1 can be computed from an equation (1-2))).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 1} \right\rbrack & \; \\{{S\begin{pmatrix}x \\y \\1\end{pmatrix}} = {\begin{pmatrix}P_{00} & P_{01} & P_{02} & P_{03} \\P_{10} & P_{11} & P_{12} & P_{13} \\P_{20} & P_{21} & P_{22} & P_{23}\end{pmatrix}\begin{pmatrix}X \\Y \\Z \\1\end{pmatrix}}} & (1)\end{matrix}$

In the perspective transformation matrix, elements P00, P01, . . . , andP23 are determined as the parameter for three-dimensional measurement ineach of the cameras A, B, and C, and the elements P00, P01, . . . , andP23 are registered in the memory 23. The completion of the registrationcan perform three-dimensional measurement to the workpiece W.

In the three-dimensional measurement processing of the embodiment, afterthe edges are extracted from the images produced by the cameras A, B,and C, each edge is divided into units called “segments” based on aconnecting point or a branch point, and the segments are correlatedamong the images. The computation is performed every combination ofcorrelated segments using the parameter, thereby deriving a set ofthree-dimensional coordinates expressing three-dimensional segment.Hereinafter the processing is referred to as “restoration ofthree-dimensional information”.

In the embodiment, for the purpose of the three-dimensional informationrestoration processing, a three-dimensional model M expressing a wholecontour shape of the workpiece W is produced as illustrated in FIG. 3.The three-dimensional model M includes a three-dimensional coordinate ofone point O (such as a gravity center) that is of a representative pointand a basic attitude (orientations of coordinate axes X, Y, and Z) thatis a reference of attitude measurement of the recognition-target objectin addition to the pieces of three-dimensional information on the pluralsegments.

In the recognition processing with the three-dimensional model M, afeature point (specifically, a branch point of the segment) in thethree-dimensional information restored by the three-dimensionalmeasurement and a feature point on the side of the three-dimensionalmodel M are correlated by a round-robin method to compute a degree ofsimilarity between both the sides. The correspondence between thefeature points is specified as a correct relationship when the degree ofsimilarity becomes the maximum. At this point, the coordinatecorresponding to the representative point O of the three-dimensionalmodel M is recognized as the position of the workpiece W. When thespecified relationship is obtained, the rotation angle of thethree-dimensional model M is recognized as the rotation angle of theworkpiece W with respect to the basic attitude indicated by thethree-dimensional model M. The rotation angle is computed in each of theaxes X, Y, and Z.

In the embodiment, a screen of FIG. 4 is displayed on the display unit25 such that a user can appropriately confirm the recognition processingresult with the registered three-dimensional model M. The confirmationscreen is horizontally divided into two parts, and an image displayregion 30 is set on the right while a manipulation region 31 is set onthe left.

An image G (hereinafter referred to as “processing object image G”) thatis produced by one of the cameras A, B, and C (at this point, thesquarely seeing camera A is selected) and used in the recognitionprocessing is displayed in the image display region 30 along with abackground region (a white portion in the image display region 30)having a predetermined extent. In FIGS. 4, 5 and 6, the letter WGdesignates the workpiece W in the processing object image G.

A graphic MG (indicating the contour shape of the workpiece W) expressedby a uniform color line indicating the three-dimensional model M is alsodisplayed in the image display region 30 while superimposed on theprocessing object image G. As illustrated in FIGS. 5 and 6, the graphicMG of the three-dimensional model is changed to a stereoscopic displayby specification of the change in direction of the line of sight withrespect to the image. Hereinafter the stereoscopic graphic MG of thethree-dimensional model is referred to as “three-dimensional modelgraphic MG”.

Input boxes 32 and sliders 33 are provided in the manipulation region 31on the left side for four kinds of numerical information of an azimuthθ, an elevation angle φ, a lateral motion amount, and a longitudinalmotion amount. The user inputs each numerical value in the input box 32,and the user changes each numerical value by a motion manipulation ofthe slider 33.

A selection box 35 is provided in an upper portion of the manipulationregion 31 in order to select the display of the image display region 30.The user can call a menu from the selection box 35 to switch the imagedisplay region to another display mode (description is omitted).

A button 34 is provided in a lower end portion of the manipulationregion 31 in order to return the display in the image display region 30to the previous state. Although not illustrated, the button 34 is set toan invalid state in the initial state screen of FIG. 4.

The azimuth θ expresses an azimuth direction of the line of sight withrespect to a reference plane of the world coordinate system. For thesake of convenience, the azimuth θ is expressed by the rotation angle ofthe processing object image G with respect to the display screen of thedisplay unit 25.

The elevation angle φ is an angle that is formed by the direction of theline of sight with respect to the reference plane of the worldcoordinate system. For example, the elevation angle φ becomes 90° whenthe image is squarely seen, and the elevation angle φ becomes 0° whenthe image is seen right beside.

From the viewpoint of the perspective transformation processing with anequation (d), the azimuth θ indicates the azimuth direction of the lineof sight in the camera coordinate system of the camera A, and theelevation angle φ indicates an angle that is formed by the direction ofthe line of sight with respect to the xy-plane (corresponding to theimaging surface) of the camera coordinate system.

The lateral motion amount and the longitudinal motion amount are used toset a disposition position of the image in the image display region 30.The lateral motion amount, the longitudinal motion amount, and theangles θ and φ are used as parameters of the perspective transformationprocessing.

The screen of FIG. 4 is displayed immediately after the recognitionprocessing is ended, the processing object image G produced by thecamera A is displayed in the image display region 30 in a usual attitude(the x-axis direction of the image correlated to the horizontaldirection of the screen while the y-axis direction is correlated to thevertical direction of the screen). The three-dimensional model graphicMG that is disposed while superimposed on the processing object image Gillustrates the state in which the three-dimensional model in which theposition and the attitude are matched with the recognition result isobserved from the line of sight of the camera A. Accordingly, when thethree-dimensional model graphic MG and the workpiece WG in theprocessing object image G are accurately aligned to each other like theexample of FIG. 4, it is believed that the recognition is accuratelyperformed with the three-dimensional model M.

The display in the image display region 30 is in the state of FIG. 4,the azimuth θ in the manipulation region 31 indicates 0°, and theelevation angle φ indicates 90°. In the embodiment, the user can freelychange the angles θ and φ to change the display of the workpiece W ofstereoscopic image.

FIG. 5 illustrates an example of the display screen in which theelevation angle φ is set to 44° while the azimuth θ is maintained at 0°.In the screen of FIG. 5, the display is changed to the display in whichthe processing object image G is observed from obliquely above, and thethree-dimensional model graphic MG is also changed to the stereoscopicimage expressing the state in which the three-dimensional model M isseen from obliquely above.

FIG. 6 illustrates the display screen in which the elevation angle φ isset to 0° while the azimuth θ is set to 45°. In the screen of FIG. 6,because of the specification that the image is rotated by 45° to beobserved right beside, the processing object image G is displayed in theone-dimensional state. The three-dimensional model graphic MG is changedto the graphic in which the three-dimensional model M is rotated by 45°from the states of FIGS. 4 and 5 to express a side face opposite theline of sight.

In the displays of FIGS. 5 and 6, the three-dimensional model graphic MGis displayed slightly above the processing object image. This is thethree-dimensional model graphic MG is compared to the processing objectimage for the sake of convenience. Alternatively, a region correspondingto a bottom surface of the three-dimensional model M may be displayedwhile correlated to the workpiece W in the processing object image.

The change of the processing object image G or the three-dimensionalmodel graphic MG is generated by the perspective transformationprocessing that is performed based on contents set in the manipulationregion 31. Because the processing object image G is two-dimensionaldata, the processing object image G is displayed as the plane even afterthe perspective transformation. On the other hand, for thethree-dimensional model graphic MG, the stereoscopic image can bedisplayed by performing the perspective transformation to thethree-dimensional information in the camera coordinate system of thecamera A, into which the transformation of the three-dimensionalinformation expressing the full-scale recognition-target object isperformed.

FIG. 7 illustrates a procedure of processing for producing thethree-dimensional model image. Four steps A, B, C, and D of theflowchart of FIG. 7 will be described in detail.

(Step A)

In the step A, the coordinate transformation of the three-dimensionalmodel registered in the memory 23 is performed based on the position androtation angle, which are recognized with respect to the workpiece W.Specifically a position deviation amount of the coordinate recognized asa representative point of the workpiece W to the coordinate registeredas the representative point O of the three-dimensional model M isdetermined in each of the axes X, Y, and Z. Elements T_(oo), T₀₁, T₀₂ .. . , and T₂₃ (see the following equation (a)) of a transformationmatrix of homogenous coordinate transformation are determined based onthe position deviation amounts and the angle (rotation angle in each ofthe axes X, Y, and Z) computed as the rotation angle of the workpiece Wto the three-dimensional model.

Then a post-transformation coordinate (Xt, Yt, Zt) is determined foreach point included in the three-dimensional model M by performing theoperation of the equation (a) to which the coordinate (XMP, YMP, ZMP) ateach point and the homogeneous transformation matrix are adapted.Hereinafter the three-dimensional model indicated by thepost-transformation coordinate (Xt, Yt, Zt) is referred to as“three-dimensional model Mt”.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 2} \right\rbrack & \; \\{\begin{bmatrix}X_{t} \\Y_{t} \\Z_{t} \\1\end{bmatrix} = {\begin{bmatrix}T_{00} & T_{01} & T_{02} & T_{03} \\T_{10} & T_{11} & T_{12} & T_{13} \\T_{20} & T_{21} & T_{22} & T_{23} \\0 & 0 & 0 & 1\end{bmatrix}\begin{bmatrix}X_{MP} \\Y_{MP} \\Z_{MP} \\1\end{bmatrix}}} & (a)\end{matrix}$

(Step B)

In the step B, the Z-coordinate Zt at each point of thethree-dimensional model Mt is corrected using the elevation angle φ.Specifically a post-correction Z-coordinate Zt′ is computed by thefollowing equation (b) to which the elevation angle φ and a Z-coordinateZ0 at the representative point of the three-dimensional model Mt areadapted. The Z-coordinate Z0 is obtained by performing the coordinatetransformation of the coordinate at the representative point O of FIG. 3in the step A.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 3} \right\rbrack & \; \\{{Zt}^{\prime} = {{\left( {{Zt} - {Z\; 0}} \right)\left( \frac{\varphi}{90} \right)} + {Z\; 0}}} & (b)\end{matrix}$

In the equation (b), when the elevation angle φ is 90°, thepost-correction Z-coordinate Zt′ is equal to the Z-coordinate Zt, thatis, the correction amount becomes zero. When the elevation angle φ is0°, the post-correction Z-coordinate Zt′ is equal to the constant valueZ0, that is, all the Z-coordinates are transformed into the constantvalue Z0 irrespective of the Z-coordinate Zt.

In the numerical range where the elevation angle φ is larger than 90°while being smaller than 0°, the post-correction Z-coordinate Zt′ comesclose to the constant value Z0 as the elevation angle φ is brought closeto 0°. The distance between the post-correction coordinates of the twoZ-coordinates having different values is shortened as the elevationangle φ is brought close to 0°.

(Step C)

In the step C, the perspective transformation of the three-dimensionalmodel Mt whose Z-coordinate is already corrected by the equation (b) isperformed into the xy-plane of the coordinate system of the camera A. Atthis point general perspective transformation will be described usingthe equation (1). The equation (1) can be dissolved into an equation(1-1) and an equation (1-2).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 4} \right\rbrack & \; \\{\begin{pmatrix}x \\y\end{pmatrix} = {\frac{1}{S}\begin{pmatrix}P_{00} & P_{01} & P_{02} & P_{03} \\P_{10} & P_{11} & P_{12} & P_{13}\end{pmatrix}\begin{pmatrix}X \\Y \\Z \\1\end{pmatrix}}} & \left( {1\text{-}1} \right) \\{S = {{P_{20}X} + {P_{21}Y} + {P_{22}Z} + P_{23}}} & \left( {1\text{-}2} \right)\end{matrix}$

Therefore, a relationship between the pre-transformationthree-dimensional coordinate (X, Y, Z) and the post-transformationtwo-dimensional coordinate (x, y) can be expressed by an equation (2) byreplacing S in the equation (1-1) by the right side of the equation(1-2).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 5} \right\rbrack & \; \\{\begin{pmatrix}x \\y\end{pmatrix} = {\frac{1}{{P_{20}X} + {P_{21}Y} + {P_{22}Z} + P_{23}}\begin{pmatrix}P_{00} & P_{01} & P_{02} & P_{03} \\P_{10} & P_{11} & P_{12} & P_{13}\end{pmatrix}\begin{pmatrix}X \\Y \\Z \\1\end{pmatrix}}} & (2)\end{matrix}$

However, because the general transformation equation expresses the statein which the stereoscopic shape indicated by the three-dimensionalinformation is imaged with the camera, even if the planes have the sameshape and the same area, the planes have the different sizes of theprojection images when the planes differ from each other in theZ-coordinate (the camera projection image becomes small as the plane isdistant from the camera, that is, as the Z-coordinate is decreased).When the perspective transformation of the projection images having thedifferent sizes is performed from the direction away from the directionorthogonal to the imaging surface, the difference in size is directlyprojected, which results in the projection image in which the shapeenvisaged from the direction of the line of sight is not correctlyexpressed.

Therefore, in the embodiment, a projection image variation amount isadjusted according to the difference of the Z-coordinate by computingthe following equation (c) in which the post-correction Z-coordinate Zt′is incorporated.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 6} \right\rbrack & \; \\{\begin{pmatrix}x \\y\end{pmatrix} = {\frac{1}{{P_{20}{Xt}} + {P_{21}{Yt}} + {P_{22}{Zt}^{\prime}} + P_{23}}\begin{pmatrix}P_{00} & P_{01} & P_{02} & P_{03} \\P_{10} & P_{11} & P_{12} & P_{13}\end{pmatrix}\begin{pmatrix}{Xt} \\{Yt} \\{Zt}^{\prime} \\1\end{pmatrix}}} & (c)\end{matrix}$

As described above, as the elevation angle φ is brought close to 0°, thepost-correction Z-coordinate Zt′ comes close to the constant value Z0,and the difference among post-correction Z-coordinate is decreased.Therefore, the projection position at each point of thethree-dimensional model Mt by the equation (c) in which thepost-correction Z-coordinate Zt′ is incorporated such that thedifference between projection positions of the different Z-coordinatesbecomes small as the elevation angle φ is brought close to 0°.

(Step D)

Each point of the three-dimensional model is projected to the xy-planeof the coordinate system of the camera A as the two-dimensionalcoordinate through the step C. In the step D, the height in which thepre-correction Z-coordinate Zt at the corresponding point of thethree-dimensional model Mt is converted into the coordinate in units ofpixels is correlated to each projected point, thereby providing thethree-dimensional coordinate of the camera coordinate system to eachpoint. The perspective transformation of the set of thethree-dimensional coordinates is performed using the pieces ofinformation set in the manipulation region 31. The specific operationalequation becomes an equation (d).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 7} \right\rbrack & \; \\{\begin{pmatrix}x^{\prime} \\y^{\prime}\end{pmatrix} = {{\begin{pmatrix}{{- \sin}\; \theta} & {\cos \; \theta} \\{\cos \; {\theta \cdot \sin}\; \varphi} & {\sin \; {\theta \cdot \sin}\; \varphi}\end{pmatrix}\begin{pmatrix}{x - {xo}} \\{{yo} - y}\end{pmatrix}} + \begin{pmatrix}{{xo} + {tx}} \\{{yo} + {ty} - {\frac{Zt}{K}\cos \; \varphi}}\end{pmatrix}}} & (d)\end{matrix}$

In the equation (d), (xo, yo) is a coordinate on the screencorresponding to an origin of the camera coordinate system, txcorresponds to the lateral motion amount specified in the manipulationregion 31, and ty corresponds to the longitudinal motion amountspecified in the manipulation region 31.

K in the equation (d) is a scale factor of the camera coordinate system(full scale per one pixel, unit is millimeter (mm)). The scale factor Kis determined as follows based on a relationship between the plane ofZ=0 of the world coordinate system and the plane of x=0 of the cameracoordinate system.

The coordinate (x0, y0) at the projection point of the origin (0, 0, 0)of the world coordinate system is computed by the following equation(2-1) in which the origin of the world coordinate system is substitutedfor the equation (2).

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 8} \right\rbrack & \; \\{\left( {{{In}\mspace{14mu} {the}\mspace{14mu} {case}\mspace{14mu} {of}\mspace{14mu} \left( {X,Y,Z} \right)} = \left( {0,0,0,} \right)} \right){\begin{pmatrix}x_{0} \\y_{0}\end{pmatrix} = {{\frac{1}{P_{23}}\begin{pmatrix}P_{00} & P_{01} & P_{02} & P_{03} \\P_{10} & P_{11} & P_{12} & P_{13}\end{pmatrix}\begin{pmatrix}0 \\0 \\0 \\1\end{pmatrix}} = {\frac{1}{P_{23}}\begin{pmatrix}P_{03} \\P_{13}\end{pmatrix}}}}} & \left( {2\text{-}1} \right)\end{matrix}$

Then the coordinate (x1, y1) at the projection point for the coordinate(1, 0, 0) and the coordinate (x2, y2) at the projection point for thecoordinate (0, 1, 0) are computed from the following equations (2-2) and(2-3) in which the coordinate (1, 0, 0) corresponding to the point 1-mmaway from the origin on the X-axis and the coordinate (0, 1, 0)corresponding to the point 1-mm away from the origin on the Y-axis aresubstituted for the equation (2), respectively.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 9} \right\rbrack & \; \\{\left( {{{In}\mspace{14mu} {the}\mspace{14mu} {case}\mspace{14mu} {of}\mspace{14mu} \left( {X,Y,Z} \right)} = \left( {1,0,0} \right)} \right){\begin{pmatrix}x_{1} \\y_{1}\end{pmatrix} = {{\frac{1}{P_{20} + P_{29}}\begin{pmatrix}P_{00} & P_{01} & P_{02} & P_{03} \\P_{10} & P_{11} & P_{12} & P_{13}\end{pmatrix}\begin{pmatrix}1 \\0 \\0 \\1\end{pmatrix}} = {\frac{1}{P_{20} + P_{23}}\begin{pmatrix}{P_{00} + P_{03}} \\{P_{10} + P_{13}}\end{pmatrix}}}}} & \left( {2\text{-}2} \right) \\\left\lbrack {{Formula}\mspace{14mu} 10} \right\rbrack & \; \\{\left( {{{In}\mspace{14mu} {the}\mspace{14mu} {case}\mspace{14mu} {of}\mspace{14mu} \left( {X,Y,Z} \right)} = \left( {0,1,0} \right)} \right){\begin{pmatrix}x_{2} \\y_{2}\end{pmatrix} = {{\frac{1}{P_{21} + P_{23}}\begin{pmatrix}P_{00} & P_{01} & P_{02} & P_{03} \\P_{10} & P_{11} & P_{12} & P_{13}\end{pmatrix}\begin{pmatrix}0 \\1 \\0 \\1\end{pmatrix}} = {\frac{1}{P_{21} + P_{29}}\begin{pmatrix}{P_{01} + P_{03}} \\{P_{11} + P_{13}}\end{pmatrix}}}}} & \left( {2\text{-}3} \right)\end{matrix}$

Then, for the camera coordinate system, a size Kx per one pixel in thedirection corresponding to the X-axis direction of the world coordinatesystem and a size Ky per one pixel in the direction corresponding to theY-axis direction of the world coordinate system are computed byperforming the equations (3) and (4) using the coordinate of eachprojection point.

It is assumed that an average value of the sizes Kx and Ky is the scalefactor K.

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 11} \right\rbrack & \; \\{{Kx} = \frac{1}{\sqrt{\left( {x_{0} - x_{1}} \right)^{2} + \left( {y_{0} - y_{1}} \right)^{2}}}} & (3) \\{{Ky} = \frac{1}{\sqrt{\left( {x_{0} - x_{2}} \right)^{2} + \left( {y_{0} - y_{2}} \right)^{2}}}} & (4)\end{matrix}$

The operation for obtaining the scale factor K can be performed afterthe calibration processing is ended.

In the equation (d), the coordinate in which the full-scale Z-coordinateZt is converted into the coordinate in units of pixels by the scalefactor K is set to each projection point. Accordingly, each pointprojected to the plane of z=0 of the coordinate system of the camera Ais moved from the plane of z=0 by a distance corresponding to Zt/K, andthe point is three-dimensionally disposed in the camera coordinatesystem to perform the perspective transformation.

Because the scale factor that is substantially equal to that of thex-coordinate or y-coordinate is set to the Z-coordinate at eachprojection point, a halting state is eliminated in an interval betweenthe projection points in the height direction, and the perspectivetransformation can be performed to the projection point in which abalance with distribution on the xy-plane is established. Thex-coordinate and y-coordinate at each projection point are adjustedaccording to the elevation angle φ by the equations (b) and (c).Therefore, even if the setting direction of the line of sight is largelydeviated from the direction in which the imaging surface is squarelyseen, the influence of the Z-coordinate on the position in which eachpoint of the three-dimensional model Mt is projected can be reduced, andthe deformation of the projection image can be prevented from beinggenerated through the second-time perspective transformation.Accordingly, the stereoscopic image in which the shape of thethree-dimensional model is expressed without a feeling of strangenesscan be produced.

FIG. 8 schematically illustrates a procedure of processing performed toone workpiece.

Referring to FIG. 8, in step S1, the three-dimensional measurement isperformed to the workpiece W. In step S2, the three-dimensionalinformation restored by the measurement is checked with thethree-dimensional model to recognize the position and rotation angle ofthe workpiece W.

In step S3, the azimuth θ and the elevation angle φ are set to aninitial value of 0°, the steps A, B, C, and D of FIG. 7 are performed tothe three-dimensional model M to produce the three-dimensional modelgraphic MG. In step S4, the produced three-dimensional model graphic MGand the processing object image G are displayed on the display unit 25while superimposed on each other. Therefore, the image in the state ofFIG. 4 is displayed on the screen of the display unit 25.

Then, until the manipulation is ended, the processing for updating thedisplays of the three-dimensional model graphic MG and processing objectimage G is performed according to the manipulation of the setting changeof the parameters for the perspective transformation (steps S5, S6, andS7).

The loop in step S6 will specifically be described.

In step S6, the steps B, C, and D of FIG. 7 are performed to thepost-coordinate-transformation three-dimensional model Mt, therebyproducing the three-dimensional model image according to the specifiedazimuth θ and elevation angle φ. That is, it is only necessary toperform the step A of FIG. 7 once after the recognition processing.Since then, the pieces of processing in the steps B, C, and D areperformed to the post-coordinate-transformation three-dimensional modelMt, which allows the three-dimensional model graphic MG to be produced.

As to the processing object image G, while the x- and y-coordinates ofeach pixel are directly applied, the z-coordinate of 0 is set to all thepixels to perform the computation similar to the equation (d). Thedisplay in the image display region 30 is updated by the display inwhich the three-dimensional model graphic MG is superimposed on thepost-transformation processing object image G.

In the procedure, after the user confirms the result and accuracy of therecognition by seeing the display of the processing object image G ofthe usual attitude and the three-dimensional model graphic MG matchedwith the processing object image G, the user can freely change theazimuth θ and the elevation angle φ to confirm the relationship betweenthe three-dimensional model and the workpiece W in the processing objectimage from various directions.

The image display of the embodiment is performed for the camera A.However, for the cameras B and C, the image produced each of the camerasis correlated to the three-dimensional model image by the similartechnique, and the image obtained by perspective transformationprocessing can be displayed from various directions. In the embodiment,at the beginning of the production of the three-dimensional modelgraphic MG, the coordinate transformation of the three-dimensional modelimage is performed based on the recognition result. When the result ofthe coordinate transformation performed through the recognitionprocessing in step S2 is stored, because of the use of the storage data,it is unnecessary to perform the coordinate transformation again.

In the embodiment, the three-dimensional information on the workpiece Wis restored by the stereo measurement using the plural cameras. Thethree-dimensional measurement method is not limited to the stereomeasurement, but a method for processing the image produced by onecamera can also be adopted. For example, a surface of the workpiece Wmay be illuminated with spot light to determine the three-dimensionalcoordinate at an illumination point of the spot light from a coordinateat a bright point in the image. Alternatively, the height of theworkpiece model is changed in various ways, the imaging is performed toproduce plural two-dimensional models, the image of the workpiece W ofthe processing object is sequentially checked with each of thetwo-dimensional models to specify the height and rotation angle of theworkpiece W, and the specification result is reflected in each featurepoint of the image to derive the plural three-dimensional coordinates.In any method, when the restored three-dimensional information ischecked with the full-scale three-dimensional model to recognize theposition and attitude of the workpiece W, the recognition result can bedisplayed by the technique similar to that of the embodiment.

1. A method for displaying recognition result of a three-dimensionalvisual sensor, the three-dimensional visual sensor including: an imagingunit that includes at least one camera; a registration unit in whichthree-dimensional information expressing a full-scale recognition-targetobject is registered as a three-dimensional model; and a recognitionunit that performs three-dimensional measurement to a recognition-targetobject whose three-dimensional model is registered in the registrationunit using an image produced by imaging of the imaging unit and checksthree-dimensional information restored by the measurement with theregistered three-dimensional model to recognize a position and anattitude of the recognition-target object, on condition that a worldcoordinate system is defined such that a distance from a reference planetraversing a direction of an optical axis of the camera in the imagingunit becomes a Z-coordinate indicating a height, wherein at least one ofthe cameras is set to a processing object, a first step, a second step,a third step, and a fourth step are performed every camera of theprocessing object, in the first step, a manipulation of specification ofa direction of a line of sight with respect to an imaging surface of thecamera being received, in the second step, a plurality of Z-coordinatesincluded in a three-dimensional model being corrected to performperspective transformation of a post-correction three-dimensional modelinto a coordinate system of the imaging surface of the camera of theprocessing object for the three-dimensional model to which coordinatetransformation is performed in a world coordinate system such that theposition and attitude of the three-dimensional model become identical tothose of the recognition-target object recognized by the recognitionunit, in the third step, perspective transformation of a point having athree-dimensional coordinate value obtained by setting a heightaccording to a corresponding pre-correction Z-coordinate into a pointhaving a three-dimensional coordinate value obtained by setting a pointincluded in an projection image of the three-dimensional model producedin the coordinate system of the imaging surface of the camera of theprocessing object by the perspective transformation processing beingperformed from the direction of the line of sight received in the firststep, in the fourth step, the projection image produced by theperspective transformation in the third step being displayed on amonitor device, and a rule of correction operation is applied to theZ-coordinate correction in the second step, in the rule of correctionoperation, a correction amount to each Z-coordinate being set to zerowhen an orthogonal relationship is established between the direction ofthe line of sight received in the first step and the imaging surface ofthe camera of the processing object, each Z-coordinate being correctedto a constant value Z0 when the direction of the line of sight becomesparallel to the imaging surface, each post-correction Z-coordinate beingbrought close to the constant value Z0 as the relationship between thedirection of the line of sight and the imaging surface becomes close toa parallel state from the orthogonal state, a difference betweenpost-correction Z-coordinates Z1′ and Z2′ of any two of Z-coordinates Z1and Z2 (Z1>Z2) being gradually decreased while a relationship of Z1′>Z2′is maintained.
 2. The method according to claim 1, wherein a step ofspecifying a size per one pixel is previously performed to the camera ofthe processing object based on a relationship between the coordinatesystem of the camera and the world coordinate system, the size per onepixel being allocated to the imaging surface in performing perspectivetransformation of a plane in which a height in a space is zero into theimaging surface of the camera of the processing object, and in the thirdstep, for each point of the projection image of the three-dimensionalmodel, the pre-correction Z-coordinate at a corresponding point of thepre-coordinate-transformation three-dimensional model is converted intothe number of pixels based on the size per one pixel specified in thestep, and the number of pixels is set to the height after theconversion.
 3. A three-dimensional visual sensor comprising: an imagingunit that includes at least one camera; a registration unit in whichthree-dimensional information expressing a full-scale recognition-targetobject is registered as a three-dimensional model; a recognition unitthat performs three-dimensional measurement to a recognition-targetobject whose three-dimensional model is registered in the registrationunit using an image produced by imaging of the imaging unit and checksthree-dimensional information restored by the measurement with theregistered three-dimensional model to recognize a position and anattitude of the recognition-target object, on condition that a worldcoordinate system is defined such that a distance from a reference planetraversing a direction of an optical axis of the camera in the imagingunit becomes a Z-coordinate indicating a height; a specifyingmanipulation receiving unit that sets at least one of the cameras to aprocessing object to receive a manipulation of specification of adirection of a line of sight with respect to an imaging surface of thecamera of the processing object; a first perspective transformation unitthat corrects a plurality of Z-coordinates included in athree-dimensional model to perform perspective transformation of apost-correction three-dimensional model into a coordinate system of theimaging surface of the camera of the processing object based on arelationship between the direction of the line of sight received by thespecifying manipulation receiving unit and the imaging surface of thecamera of the processing object, the three-dimensional model to whichcoordinate transformation is performed in a world coordinate system suchthat the position and attitude of the three-dimensional model becomeidentical to those of the recognition-target object recognized by therecognition unit; a second perspective transformation unit that performsperspective transformation of a point having a three-dimensionalcoordinate value obtained by setting a height according to acorresponding pre-correction Z-coordinate into a point having athree-dimensional coordinate value obtained by setting a point includedin an projection image of the three-dimensional model produced in thecoordinate system of the imaging surface of the camera of the processingobject by the perspective transformation processing performed throughthe first perspective transformation unit from the direction of the lineof sight received in the specifying manipulation receiving unit; and adisplay control unit that displays the projection image produced throughthe perspective transformation performed by the second perspectivetransformation unit on a monitor device, wherein a rule of correctionoperation is applied to the Z-coordinate correction in the firstperspective transformation unit, in the rule of correction operation, acorrection amount to each Z-coordinate being set to zero when anorthogonal relationship is established between the direction of the lineof sight received in the first step and the imaging surface of thecamera of the processing object, each Z-coordinate being corrected to aconstant value Z0 when the direction of the line of sight becomesparallel to the imaging surface, each post-correction Z-coordinate beingbrought close to the constant value Z0 as the relationship between thedirection of the line of sight and the imaging surface becomes close toa parallel state from the orthogonal state, a difference betweenpost-correction Z-coordinates Z1′ and Z2′ of any two of Z-coordinates Z1and Z2 (Z1>Z2) being gradually decreased while a relationship of Z1′>Z2′is maintained.